Build in debug mode for PRs by timsaucer · Pull Request #1375 · apache/datafusion-python

timsaucer · 2026-02-08T14:14:42Z

This PR is to improve our CI performance in a couple of ways

For PRs we can build in debug mode instead of release mode. For RC tags and pushes to main we still run in release mode.
This gives better debug if there are failures in CI and faster builds.
We will still catch any errors generated in release mode when the merge to main occurs.
At the same time, doing some minor cleanup of jobs that may no longer be necessary.
Upload only one artifact in debug mode, the linux x86 wheel used for pytests
Avoids rebuilding multiple times in test
Overall cuts down CI by about 30 minutes

… releases

…ding the wheel artifacts from build stage

…oml file

…pushes to main and release or candidate tags.

…s erasing the other wheel

.github/workflows/build.yml

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Copilot

Pull request overview

This PR refactors CI to use reusable build/test workflows and to run PR builds in debug mode for faster feedback while keeping release-mode builds for main and tags.

Changes:

Split CI into reusable workflows (build.yml, test.yml) and added dedicated entry workflows for PRs (ci.yml) and releases (release.yml).
Updated test workflow to install and test against the pre-built Linux wheel artifact instead of rebuilding during tests.
CI/config formatting cleanups across TOML files and adjusted license generation to rely on cargo-license.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pyproject.toml	Formatting cleanup and a dependency list syntax fix (trailing comma).
examples/datafusion-ffi-example/Cargo.toml	TOML formatting cleanup for `pyo3` features list.
examples/datafusion-ffi-example/.cargo/config.toml	TOML formatting cleanup for `rustflags`.
dev/create_license.py	Switches license generation to call `cargo-license` directly.
.github/workflows/test.yml	Converts to reusable workflow and installs pre-built wheel artifacts for tests.
.github/workflows/release.yml	Adds release entry workflow calling reusable build/test in release mode.
.github/workflows/ci.yml	Adds PR entry workflow calling reusable build/test in debug mode.
.github/workflows/build.yml	Converts to reusable workflow; adds linting jobs and reworks wheel build/artifact behavior.
.cargo/config.toml	TOML formatting cleanup for `rustflags`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-09T21:16:57Z

.github/workflows/build.yml

+  # ============================================
+  # Build - Linux ARM64
+  # ============================================
+  build-manylinux-aarch64:


inputs.run_wheels is passed by callers (false for PRs) but the non-Linux-x86_64 wheel build jobs still execute in debug mode and only skip uploading artifacts. This likely negates a chunk of the intended PR CI speedup. Consider gating build-manylinux-aarch64 and build-python-mac-win at the job level with if: inputs.run_wheels (or if: inputs.build_mode == 'release') so PRs only build the single Linux x86_64 wheel needed for pytest.

Suggested change

build-manylinux-aarch64:

build-manylinux-aarch64:

if: inputs.run_wheels

Copilot · 2026-02-09T21:16:58Z

.github/workflows/build.yml

+      - name: Archive wheels
+        uses: actions/upload-artifact@v6
+        if: inputs.build_mode == 'release'


inputs.run_wheels is passed by callers (false for PRs) but the non-Linux-x86_64 wheel build jobs still execute in debug mode and only skip uploading artifacts. This likely negates a chunk of the intended PR CI speedup. Consider gating build-manylinux-aarch64 and build-python-mac-win at the job level with if: inputs.run_wheels (or if: inputs.build_mode == 'release') so PRs only build the single Linux x86_64 wheel needed for pytest.

Copilot · 2026-02-09T21:16:58Z

.github/workflows/build.yml

  build-python-mac-win:
-    needs: [generate-license]
-    name: Mac/Win
+    needs: [generate-license, lint-rust, lint-python]
+    name: macOS arm64 & Windows


inputs.run_wheels is passed by callers (false for PRs) but the non-Linux-x86_64 wheel build jobs still execute in debug mode and only skip uploading artifacts. This likely negates a chunk of the intended PR CI speedup. Consider gating build-manylinux-aarch64 and build-python-mac-win at the job level with if: inputs.run_wheels (or if: inputs.build_mode == 'release') so PRs only build the single Linux x86_64 wheel needed for pytest.

Copilot · 2026-02-09T21:16:58Z

.github/workflows/build.yml

      - name: Archive wheels
        uses: actions/upload-artifact@v6
+        if: inputs.build_mode == 'release'


inputs.run_wheels is passed by callers (false for PRs) but the non-Linux-x86_64 wheel build jobs still execute in debug mode and only skip uploading artifacts. This likely negates a chunk of the intended PR CI speedup. Consider gating build-manylinux-aarch64 and build-python-mac-win at the job level with if: inputs.run_wheels (or if: inputs.build_mode == 'release') so PRs only build the single Linux x86_64 wheel needed for pytest.

Copilot · 2026-02-09T21:16:58Z

.github/workflows/build.yml

+        run: uv run --no-project maturin build --release --strip --features substrait
+
+      - name: Build Python package (debug mode)
+        if: inputs.build_mode != 'release'
+        run: uv run --no-project maturin build --features substrait


The comment says clippy features must match maturin exactly, but clippy runs with --features protoc,substrait while maturin builds use --features substrait only. This is a concrete mismatch that can cause clippy to validate a different codepath than what is shipped on macOS/Windows. Align the features (either add protoc to maturin builds here, or remove it from clippy) and keep it consistent with the Linux wheel builds if protoc is required there.

Suggested change

run: uv run --no-project maturin build --release --strip --features substrait

- name: Build Python package (debug mode)

if: inputs.build_mode != 'release'

run: uv run --no-project maturin build --features substrait

run: uv run --no-project maturin build --release --strip --features protoc,substrait

- name: Build Python package (debug mode)

if: inputs.build_mode != 'release'

run: uv run --no-project maturin build --features protoc,substrait

Copilot · 2026-02-09T21:16:59Z

dev/create_license.py

 data = subprocess.check_output(
    [
-        "cargo",
-        "license",
+        "cargo-license",
        "--avoid-build-deps",
        "--avoid-dev-deps",
        "--do-not-bundle",


Calling cargo-license directly now requires the binary to already be available on PATH. Since this script may be run outside CI, failures will be less clear than before. Consider adding a small preflight check (e.g., via shutil.which('cargo-license')) and raising a targeted error message instructing how to install it, or falling back to invoking it via cargo license so it works when installed as a cargo subcommand.

Copilot · 2026-02-09T21:16:59Z

.github/workflows/test.yml

+        run: |
+          set -x
+          uv venv
+          # Install documentation dependencies


The comment says "Install documentation dependencies" but the command shown is a generic dev sync without selecting a docs group (contrast with build.yml which uses --group docs). Either adjust the comment to reflect what's actually installed, or include the appropriate dependency group if docs deps are intended here.

Suggested change

# Install documentation dependencies

# Install development dependencies

timsaucer added 27 commits February 8, 2026 09:12

First draft of running debug mode for PRs and release mode for main &…

887c303

… releases

Update paths

2917c6b

Change install command for taplo

7f4e1f2

install protoc

34e81a3

taplo fmt

7ffe145

Working through CI build issues

1de00cc

More CI issues

49f949e

do not build taplo, just download it

bf0bb97

Try only running clippy when we can reuse build artifacts

a7ed2f6

Try removing unnecessary installs during build

1d8949d

Don't build cargo-license

fe57e0d

Add back in uv sync so we can run maturin

632bd8a

minor: name casing

b3e905a

Fix path for wheels

4097003

More CI updates, but expect pytest to fail until we switch to downloa…

6f0d76b

…ding the wheel artifacts from build stage

Correct error in yml file. Rename to match other file extensions

8e872d1

Download wheel from build stage for testing

c08a788

For CI tests move into test directory to avoid picking up pyproject.t…

b8e0e01

…oml file

Do not upload artifacts not used in testing during debug builds

0cc6642

Do not attempt to use local python path for tests

c9ff41f

Bump manylinux version

b83bed1

Do not run release flow for branches named branch-*. Only run it for …

aa88d65

…pushes to main and release or candidate tags.

Build FFI test code in build stage so we only build it once

a8fa0fb

We need both wheels to be in the dist folder and the maturin action i…

efa3ba7

…s erasing the other wheel

We now have two wheels that need to be installed instead of just one

c14a3a7

Make a minor change to restart CI

cd5c4d6

tests will need both wheels also

03f4d15

timsaucer marked this pull request as ready for review February 9, 2026 20:34

martin-g reviewed Feb 9, 2026

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

Update .github/workflows/build.yml

facd83e

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

timsaucer requested a review from Copilot February 9, 2026 21:15

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build in debug mode for PRs#1375

Build in debug mode for PRs#1375
timsaucer wants to merge 28 commits intoapache:mainfrom
timsaucer:feat/improve-ci-time

timsaucer commented Feb 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Copilot AI Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	build-manylinux-aarch64:
	build-manylinux-aarch64:
	if: inputs.run_wheels

	# Install documentation dependencies
	# Install development dependencies

Conversation

timsaucer commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

timsaucer commented Feb 8, 2026 •

edited

Loading